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Our  goal  has  been  to  understand  visual  cortical  function  using  psychophysical 
techniques.  In  particular,  we  examined  visual  attention,  visual  search,  visual  surface 
representation  and  color  filling  in.  In  visual  attention,  we  found  evidence  to  suggest  two 
components  of  focal  attention,  a  transient  and  a  sustained  process.  We  also  provided  evidence 
for  the  role  of  attention  in  saccadic  eye  movements  by  showing  rapid  deployment  of  attention  in 
the  gap  paradigm.  In  visual  search,  we  found  a  particular  situation  where  increasing 
distractor  number  led  to  decreasing  reaction  times,  suggesting  a  different  role  for  attention  in 
such  taste.  In  visual  surface  representation,  we  outlined  the  role  of  surface  encoding  (border 
ownmhip,  modal  and  amodal  completion,  transparency)  in  many  visual  taste.  From  these 
studies  we  postulated  the  principle  of  generic  image  sampling,  a  hypothesis  which  provides  a 
geometric  tool  to  understand  visual  surface  learning.  We  also  have  investigated  the  perception 
of  dep^  from  unpaired  points  (DaVinci  stereopsis),  showing  that  such  points  lead  to  depth  and 
subjective  contours.  In  color  filling  in,  we  have  explored  spatial-temporal  dynamics  and  have 
created  a  neural  model. 
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During  the  recently  completed  grant  period,  considerable  progress  has  * 

been  made  in  a  variety  of  project  areas.  We  divide  our  research  i 

accomplishments  into  several  broad  areas  :  Visual  surface  representation, 
visual  attention  and  visual  search,  plus  a  smaller  project  on  brightness  and 
color  filling  in  as  well  as  tilt-after-effects  with  real  and  subjective  contours.  ^ 

We  briefly  summarize  progress  in  each  area  in  turn. 


Visual  surface  representation: 

The  main  bulk  of  this  work  on  surface  representation  has  been  done  in 
collaboration  with  Shinsuke  Shimojo,  originally  a  post-doctoral  fellow  in  my 
laboratory  in  San  Francisco  and  now  Associate  Professor  of  Psychology, 
University  of  Tokyo.  Our  joint  work  deals  with  three  related  areas,  all 
concerned  with  the  multiple  representation  of  surfaces.  Our  findings  were 
published  as  7  refereed  papers  and  two  book  chapters,  one  honoring  Torsten 
Wiesel  in  a  Festschrift  volume,  plus  a  summary  chapter  appearing  in  the 
recent  Cold  Spring  Harbor  Symposium  on  Quantitative  Biology,  volume  55; 
the  Brain. 

First,  we  covered  topics  in  the  completion  of  "invisible"  surfaces 
behind  occluders  (Nakayama,  Shimojo,  and  Silverman,  1989;  Shimojo, 
Silverman  and  Nakayama,  1989;  Shimojo  and  Nakayama,  1990,  Nakayama 
and  Shimojo,  1991).  In  these  papers,  using  both  with  static  and  moving 
images,  we  addressed  the  issue  of  boundedness  of  surfaces  and  the 
termination  of  lines.  We  asserted  that  a  bounded  surface  in  an  image  could 
be  either  surrounded  by  an  intrinsic  border  (reflecting  the  outline  of  the 
surface  itself)  or  it  could  be  bounded  as  a  result  of  occlusion.  Likewise  for 
lines.  A  line  can  terminate  because  it  really  does  come  to  an  end  in  the  real 
world,  or  it  could  terminate  because  of  occlusion.  In  a  similar  vein,  such  a 
line  can  have  either  an  intrinsic  or  extrinsic  terminator.  To  deal  with  such  a 
possibility,  we  hypothesized  that  the  visual  system  treats  borders  separating 
one  image  region  and  another  in  a  one-sided  way  (Nakayama  and  Shimojo, 


1991).  What  we  mean  is  that  ordinarily  the  border  will  "belong"  to  either 

one  region  or  the  other,  not  both.  Furthermore,  in  our  experiments,  we  show  (O  i 

that  depth  plays  a  decisive  role  in  determining  which  region  "owns"  the  UDl 

border.  Ordinarily,  it  belongs  to  the  closer  surface.  The  assignment  of  border  LO  m 

ownership  is  decisive  in  a  number  of  visual  tasks,  in  perceptual  grouping  and 

pattern  recognition  (Nakayama,  Shimojo,  and  Silverman,  1989),  in  solving  I  M 

the  aperture  problem  for  motion  (Shimojo,  Silverman,  and  Nakayama,  1989)  J 


and  for  determining  motion  correspondence  in  apparent  movement  displays 
(Shimojo  and  Nakayama,  1990). 

We  show  that  if  an  image  region  does  not  "own"  its  bounding  contour, 
it  is  effectively  unbounded  there  and  can  thus  link  up  with  other  unbounded 
image  regions  to  aid  in  the  perception  of  surfaces  which  are  literally  hidden 
behind  other  surfaces.  This  completion  of  surfaces  behind  closer  surfaces 
(called  amodal  completion  by  Michotte  and  Kanisza)  powerfully  effects  a 
variety  of  seemingly  primitive  visual  functions.  As  such,  we  suggest  that  it  is 
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implemented  relatively  early  in  visual  processing.  In  fact,  we  suggest  that  it 
is  not  inconceivable  that  it  could  be  mediated  by  combinations  of  signals 
arising  from  cell  types  in  the  striate  cortex  with  known  receptive  field 
properties.  Thus,  to  encode  line  continuation  or  termination  of  lines  in  the 
real  world,  we  hypothesize  that  hypercomplex  cells  (end  stopped  cells  )  could 
be  gated  by  disparity  sensitive  cells  coding  depth  in  the  region  of  image  line 
termination  (see  Shimojo  and  Nakayama,  1988). 

Second,  we  have  explored  the  problem  of  occlusion  in  relation  to 
binocular  vision.  Capitalizing  on  the  fact  that  there  is  a  very  specific  pattern 
of  unpaired  monocular  points  in  the  binocular  image,  we  have  outlined  and 
systematized  a  number  of  important  phenomenon  which  we  label  under  the 
rubric  of  DaVinci  stereopsis.  We  summarize  these  results  in  a  number  of 
publications  (Shimojo,  Silverman,  and  Nakayama,  1988;  Shimojo  and 
Nakayama,  1989;  Nakayama  and  Shimojo,  1990;  Nakayama  and  Shimojo, 
1991).  In  this  work,  we  show  that  instead  of  seeing  binocular  rivalry  when 
viewing  these  unmatched  points,  our  visual  system  is  remarkably  adaptive, 
providing  us  with  reliable  depth  from  such  points.  Even  more  surprising  is 
the  fact  that  the  presence  ot  such  unpaired  points  can  give  rise  to  the 
formation  of  subjective  occluding  contours  and  subjective  occluding  surfaces 
(Shimojo  and  Nakayama,  1991).  In  all  our  experimental  work,  we  find  that 
perception  is  critically  dependent  on  which  eye  receives  the  unpaired  points. 
For  example,  lack  of  rivalry  and  appropriate  perceived  depth  of  these  points 
requires  that  only  "ecologically  valid"  configurations  of  disparity  and 
unpaired  points  will  be  used.  In  addition,  the  location  of  subjective 
occluding  contours  is  critically  dependent  on  which  eye  is  stimulated;  it 
appears  to  the  immediate  left  of  right-eye-only  points  and  to  the  immediate 
right  of  left-eye-only  points.  Because  eye-of-origin  information  is  only 
explicit  in  VI  and  disappears  in  higher  extra-striate  cortical  areas  (Hubei  and 
Wiesel,  1968;  van  Essen,  1985),  it  suggests  that  the  the  processing  of  this  form 
of  occlusion  is  mediated  either  in  the  striate  cortex  or  in  areas  receiving  direct 
projections  from  striate  cortex.  The  results  also  reinforce  the  view  that 
subjective  contours  are  the  result  of  an  early  cortical  process,  occuring  at  least 
by  V2  (see  von  der  Heydt  et  at,  1983).  We  also  argue  that  DaVinci  stereopsis  is 
likely  to  be  learned,  built  up  over  many  exposures  of  unpaired  points  and 
occluding  contours. 

Third,  we  have  explored  and  are  continuing  to  explore  the  curious 
emergence  of  perceived  transparency  in  a  number  of  simple  untextured 
stereograms.  In  Nakayama,  Shimojo,  and  Ramachandran  (1991)  we  outlined 
the  conditions  under  which  perceived  transparency  is  triggered  as  well  as 
showing  that  when  it  is  triggered,  seemingly  more  primitive  features  can  be 
drastically  altered.  Thus  when  perceived  transparency  is  triggered,  we  see 
color  spreading  into  otherwise  uncolored  areas,  subjective  contours  appearing 
in  some  sections  of  the  image  and  removed  in  others,  depth  of  regions  is 
radically  altered,  and  object  shapes  are  dramatically  changed  because  of  the 
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reversal  of  figure  and  ground.  We  hypothesize  that  there  are  3  critical 
variables  important  in  the  perception  of  transparency.  These  are  depth, 
luminance  and  the  presence  of  collinearly. 

Visual  attention  and  visual  search 

Work  in  this  area  has  been  done  in  two  separate  collaborative  efforts, 
one  with  Manfred  Mackeben,  the  other  with  Mary  Bravo.  Each  is  described  in 
turn. 

Cued  attention  experiments 

In  our  experiments  on  spatially  directed  visual  attention  done  in 
collaboration  with  M.  Mackeben  at  Smith  Kettlewell  Eye  Research  Institute, 
we  elucidated  a  transient  and  a  sustained  component  of  visual  attention 
using  a  discrimination  paradigm.  We  found  a  large  difference  in  visual 
performance  as  a  function  of  cue  lead  time  for  visual  discrimination  tasks 
where  the  target  could  not  act  as  its  own  cue  (see  Nakayama  and  Mackeben, 
1989).  Our  conclusion  was  that  there  were  two  distinct  components  of  focal 
visual  attention,  a  sustained  component  more  likely  to  be  under  voluntary 
control  and  a  transient  component,  more  likely  to  be  shaped  by  earlier  visual 
properties.  A  series  of  control  experiments  revealed,  however,  that  the 
transient  component  could  not  be  attributed  to  any  known  property  of  the 
visual  system,  such  as  transient  cells,  or  the  like. 

More  recently,  we  completed  an  additional  study  on  "express 
attentional"  shifts.  Our  goal  was  to  test  Burkart  Fischer's  hypothesis  that 
express  saccades  are  caused  by  a  rapid  deployment  of  attention  to  the  target 
site  (Fischer,  1987) .  Using  the  "gap"  paradigm  which  favors  the  appearance  of 
express  saccades,  we  predicted  and  found  that  the  attentional  shift  to  a 
peripheral  target  was  much  more  rapid  for  gap  times  comparable  to  those 
which  elicit  express  saccades.  Control  experiments  ruled  out  competing 
explanations  and  supported  the  view  that  the  "gap"  paradigm  (as  used  by 
Fischer  and  others),  facilitates  an  attentional  shift  to  the  periphery  because  it 
enables  an  attentional  disengagement  from  the  fovea.  As  such,  our  study 
provided  important  missing  data  to  link  attention  to  saccadic  eye  movements 
This  paper  (Mackeben  and  Nakayama,  1992)  has  just  been  accepted  for  Vision 
Research. 

Visual  search  experiments 

In  visual  search  experiments,  conducted  largely  in  collaboration  with 
Dr.  Mary  Bravo,  our  starting  point  was  to  think  about  the  allocation  of  focal 
attention  in  visual  search  displays  Seriously  considering  various  theories  of 
attentional  guidance  (Koch  and  Ullman,  1985;  Treisman,  1985;  Julesz,  1985), 
we  made  a  somewhat  counterintuitive  set  of  predictions.  Under  some 
circumstances,  we  predicted  that  reaction  times  should  decrease  with 
increasing  numbers  of  distractors  in  a  visual  search  task,  rather  than  increase 
as  has  generally  been  the  case.  Our  approach  was  to  use  tasks  where 
attentional  focussing  to  the  target  was  required,  asking  the  observer  to  make  a 
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speeded  forced-choice  discrimination  of  a  non-cue  feature  of  the  target. 
Because  this  feature  was  also  present  in  the  distractors,  it  required  that  the 
observer  focus  attention  on  the  target.  For  most  of  the  experiments,  we  used 
colored  cues  and  had  the  observer  make  a  discrimination  as  to  the  shape  of 
the  odd  colored  target.  Two  primary  paradigms  were  used.  In  one  case,  the 
colors  of  the  distractor  could  reverse  from  trial  to  trial  (the  mixed  case).  In  the 
other  case,  target  and  distractor  color  remained  constant  throughout  a  block 
of  trials.  As  predicted  from  a  consideration  of  "top  down"  vs  "bottom  up" 
theories,  only  in  the  mixed  case  did  reaction  times  decrease  with  increasing 
numbers  of  distractors.  In  addition,  when  using  the  more  usual  present-vs- 
absent  task  employed  by  others,  the  reaction  time  vs  distractor  number 
function  was  also  flat,  as  expected.  We  draw  a  number  of  rather  somewhat 
unusual  conclusions  from  this  data.  Most  important,  and  contrary  to  the 
conclusions  drawn  by  others,  we  hypothesize  that  ordinary  visual  search  tasks 
do  not  reveal  properties  of  preattentive  vision  exclusively,  but  are  more 
likely  to  reflect  post  attentive  processing,  (this  paper  will  shortly  appear  in 
Perception  and  Psychophysics). 

In  addition  to  this  empirical  work  just  described,  we  also  mention  our 
speculative  theory  entitled  the  "Iconic  bottleneck  (Nakayama,  1991), 
written  in  1986.  This  was  outlined  in  the  previous  proposal  and  is  yet  to 
appear  in  print,  delayed  as  a  book  chapter  edited  by  Colin  Blakemore. 

Color  filling  in 

In  collaboration  with  Dr.  Michael  Paradiso,  a  post-doctoral  fellow  in 
my  lab  in  San  Francisco,  we  have  completed  both  empirical  and  theoretical 
studies  of  color  filling-in.  Our  first  goal  was  to  see  whether  filling-in, 
hypothesized  to  occur  in  stabilized  vision  and  in  patients  with  retinal  lesions, 
might  also  be  seen  in  normal  everyday  vision.  Our  stimulus  was  a 
homogeneous  disk,  presented  briefly.  When  viewed  in  isolation,  it  appeared 
as  just  that,  a  homogeneous  disk.  However,  when  it  was  followed  by  various 
patterns,  either  in  the  same  or  the  other  eye,  a  dramatical  reduction  in 
brightness  in  the  center  of  the  disk  could  be  seen.  Our  original  working 
hypothesis  was  that  a  hypothetical  color-filling  process  had  been  interrupted. 
By  varying  the  distance  of  a  circular  pattern  following  the  disk  anr  its  onset 
relative  to  the  disk,  we  were  able  to  characterize  the  process  as  rae  of  spatial 
propagation  with  a  relatively  high  velocity.  Early  modeling  of  the 
phenomenon  outlined  a  process  of  a  filling  in,  more  akin  to  fluid  originating 
at  contour  boundaries  and  rushing  in  to  the  unfilled  regions.  This  model  was 
presented  as  Paradiso  and  Nakayama  (1989).  An  additional  set  of 
psychophysical  experiments,  however,  suggested  that  such  a  model  may  be 
inadequate.  In  these  later  experiments,  we  constructed  a  circular  mask  with 
very  large  gaps  in  the  circle's  perimeter,  ones  that  should  be  essentially 
transparent  to  brightness  filling.  Surprisingly,  such  masks  also  had  a  great 
effect  in  preventing  color  filling,  something  that  would  not  be  easily  expected 
if  we  took  the  filling  idea  as  being  analogous  to  fluid  flow.  As  a  result  of  these 
last  experiments,  we  have  now  constructed  a  very  different  type  model. 


inspired  more  by  concepts  of  "smoothness"  rather  than  fluid  flow.  Such  a 
filling  process  might  be  understood  more  as  a  fabric  whose  edges  are  quickly 
raised  by  the  presentation  of  a  disk  and  the  middle  of  which  is  raised  quicldy 
and  ballisticai;^  in  response  to  the  elevation  of  the  perimeter.  The  action  of 
the  pattern  mask,  therefore,  is  to  tack  down  the  fabric,  preventing  it  from 
rising.  The  psychophysical  findings  are  presented  Paradiso  and  Nakayama( 
1991).  In  addition,  the  revised  model  was  presented  at  the  1990  ARVO 
meeting  and  a  manuscript  of  this  theoretical  work  is  in  preparation. 

Interaction  of  real  and  subjective  contours 

Also  in  collaboration  with  Dr.  Paradiso  and  Shimojo,  we  found  that 
there  was  an  asymmetric  interaction  between  real  and  subjective  contours 
when  studying  the  tilt  after  effect.  We  found  that  adapting  to  subjective 
contours  could  only  alter  perceived  orientation  of  subjective  contours; 
whereas,  the  adaptation  to  real  contours  could  alter  the  perceived  orientation 
of  both  real  and  subjective  contours.  Furthermore,  we  found  that  the 
interocular  transfer  of  this  after-effect  was  much  higher  for  subjective 
contours  than  for  real  contours.  Both  findings  taken  together  indicate  that 
subjective  contours  are  encoded  later  than  real  contours  in  cortical  areas 
where  the  proportion  of  binocular  cells  is  higher.  Yet,  their  critical  and 
similar  dependence  on  orientation  differences  suggests  that  illusory  contours 
are  subject  to  the  same  interactions  as  cortical  neurons  coding  real  edges.  As 
such,  it  suggests  that  the  locus  of  subjective  contour  representation  must  be 
sufficiently  low  to  be  governed  by  the  same  rules  as  orientation  tuned  cells. 
This  contrasts  sharply  to  earlier  views  suggesting  that  illusory  contours  are 
higher  order  cognitive  processes  because  ordinarily  we  wouldn't  expect  such 
higher  order  processes  to  show  orientation  tuning.  This  paper  has  been 
published  as  Paradiso,  Shimojo,  and  Nakayama  (1989). 


Published  Papers 

1.  Nakayama,  K.  and  Shimojo.  S.  "Towards  a  neural  understanding  of  visual 
surface  representation".  In  The  Brain:  volume  55,  Proceedings  Cold  Spring 
Harbor  Symposium  on  Quantitative  Biology,  eds.  T.  Sejnowski,  E.  Kandel, 
C.F.  Stevens,  J.D.  Watson. 

2.  Shimojo.  S.  and  Nakayama,  K.  Real  world  occlusion  constraints  and 
binocular  rivalry.  Vision  Res.  30,  69-80,  1990. 

3.  Shimojo,  S.,  Silverman,  G.H.  and  Nakayama,  K.  An  occlusion-related 
mechanism  of  depth  perception  based  on  motion  and  interocular  sequence. 
Nature,  333,  265-268, 1988. 

4. Nakayama,  K.  and  Shimojo,  K.  DaVinci  stereopsis:  depth  and  subjective 
occluding  contours  from  unpaired  image  points.  Vision  Res.  30,  1181-1825, 
1990. 


pages 


Ken  Nlakavama 


5.  Nakayama,  K.,  Shimojo,  S.,  and  Silverman,  G.H.  Stereoscopic  depth:  its 
relation  to  image  segmentation,  grouping,  and  the  recognition  of  occluded 
objects.  Perception,  18,  55-68, 1989. 

6.  Shimojo,  S.,  Silverman,  G.H.  and  Nakayama,  K.  Occlusion  and  the 
solution  to  the  aperture  problem  for  motion.  Vision  Res.  29,  619-626,  1989. 

7.  Shimojo,  S.  and  Nakayama,  K.  Amodal  representation  of  occluded 
surfaces:  role  of  invisible  stimuli  in  apparent  motion  correspondence. 
Perception  19,  285-299,  ,  1990 

8.  Nakayama,  K.,  Shimojo,  S.,  and  Ramachandran,  V.S.  Transparency: 
relation  to  depth,  subjective  contours,  and  neon  color  spreading.  Perception 
19, 497-513, 1991. 

9.  Nakayama,  K.  and  Mackeben,  M.  Sustained  and  transient  components  of 
focal  visual  attention.  Vision  Res.  29,  11,  1631-1647,  1989. 

10.  Paradise,  M.,  Shimojo,  S.  and  Nakayama,  K.  Subjective  contours,  tilt 
aftereffects,  and  visual  cortical  organization.  Vision  Res.  29,  1205-1213,  1989. 

11.  Paradise,  M.A.  and  Nakayama,  K.  Brightness  perception  and  filling-in 
Vision  Research  31,  1221-1236,  1991. 

12  Nakayama,  K.  and  Shimojo,  S.  Intermediate  and  higher  order  aspects  of 
motion  processing.  In.  Proceedings  of  the  retina  research  foundation 
symposium,  volume  2,  Neural  mechanisms  of  visual  perception,  D.  M-K 
Lam  and  C.B.  Gilbert,  eds)  Portfolio  publishing  company  ,  the  Woodlands, 
p.  281-296, 1989. 

13.  Nakayama,  K.  The  iconic  bottleneck  and  the  tenuous  link  between  early 
visual  processing  and  perception.  In,  Vision:  coding  and  efficiency,  C. 
Blakemore,  (Ed),  Cambridge  University  Press,  pp  411-422,  1990. 

Papers  in  press  or  under  review 

14.  Mackeben,  M.  and  Nakayama,  K.  Express  attentional  saccades  (Vision 
Research,  In  Press) 

15.  Bravo,  M.  and  Nakayama,  K.  The  role  of  attention  in  differeng  forms  of 
search  tasks  (Perception  and  Psychophysics,  in  press). 

16.  Nakayama,  K.  and  Shimojo,  S.  Visual  Surface  Perception.  Invited 
manuscript  for  Science,  (under  review) 

17.  Shimojo,  S.  and  Nakayama,  K.  Interocularly  Unpaired  Zones  Escape  Local 
Binocular  Matching,  Nature  (under  review). 


aageii 


Ken  Nakayama 


